Human pursuit system, human pursuit apparatus and human pursuit program

ABSTRACT

A human pursuit system includes a plurality of cameras, shooting directions of which are directed toward a floor, are installed on a ceiling, a parallax of an object reflected in an overlapping image domain is calculated on the basis of at least a portion of the overlapping image domain where images are overlapped among shot images shot by the plurality of cameras, the object equal to or greater than a threshold value predetermined by the calculated parallax is detected as a human, a pattern image including the detected human object is extracted, and a pattern matching is applied to the extracted pattern image and the image shot by the camera to thereby pursue a human movement trajectory.

BACKGROUND OF THE INVENTION

The present invention relates to a human pursuit system, a human pursuitapparatus and a human pursuit program, and particularly to a humanpursuit technique capable of providing on a ceiling in a building, andsuitable for measuring human movement trajectories in the buildings orstores.

A measurement for the human movement trajectories is sometimes performedfor purposes of security measures, traffic counts, customer preferencesurveys in stores, etc.

For example, JP-A-2004-348618 discloses a technique to obtain humanmovement trajectories in a store, in which a number of cameras installedin the store take pictures of customers who are shopping and movingtherein, features such as gender, age, costume, etc. are recognized forevery frame of images relative to the pictures, an identificationprocessing is carried out by using the recognized feature to identifywith images one another taken by the other cameras, and a human movementtrajectory in the store is then obtained from the processing. Further,JP-A-2006-209585 discloses a technique for using a face-recognitionprocessing to identify human objects.

SUMMARY OF THE INVENTION

However, the cameras are often installed close to a ceiling at a smalldepression angle for a purpose of making an open space covered by acamera broad, in related techniques. For example, in the case wherethere are a number of blocking objects in a small store where customersare crowded in an open space, the movement trajectory of human cannot bepursued accurately in the store since the human is hidden behind goodsor others.

According to the foregoing related techniques, the identificationprocessing for human object is performed by using the featurescontaining the gender, age costume, etc. and a result of theface-recognition processing, obtained from the human images. However, itis necessary to take pictures in high resolution for a purpose ofextracting the face-recognition processing and an amount of features.Therefore, it is also necessary to use high performance cameras or anumber of cameras. Besides, a pursuing accuracy for the human movementtrajectory is deteriorated since the identification processing causes anerror in response to a face direction and a position of the humanobject.

An object of the invention is to provide a human pursuit system, a humanpursuit apparatus, and a human pursuit program, which is able to pursuea human movement trajectory in high accuracy even though blockingobjects are resided in an open space.

According to an aspect of the invention, a human pursuit systemincludes: a plurality of cameras installed on a ceiling and shootingdirections of these directed toward a floor; means that calculates aparallax of an object reflected in an overlapping image domain inaccordance with at least a portion of the overlapping image domain whereimages are overlapped, among shot images shot respectively by theplurality of cameras; means that detects the object, as a human, whenthe calculated parallax is greater than a predetermined threshold value;means that extracts a pattern image containing the detected human; andmeans that pursues a trajectory on which the human moves, by a patternmatching of the extracted pattern image with the shot image by thecamera.

According to another aspect of the invention, the human pursuit systemfurther includes calculation means that calculates a position coordinateof the human residing on the floor in accordance with a position of thepattern image containing the detected human and the parallax calculatedfor the human.

According to still another aspect of the invention, the human pursuitsystem further includes storage means that stores the extracted patternimage for every detected human, and update means that updates thepattern image stored in the storage means in accordance with a result ofthe pattern matching.

According to still another aspect of the invention, the storage means inthe human pursuit system stores the position coordinate of the patternimage on the floor in association with the pattern image, and the updatemeans therein updates the pattern image in accordance with an imagedomain of the human detected from the overlapping image domain in thecase where the stored pattern image is contained in the overlappingimage domain.

According to still another aspect of the invention, the update means inthe human pursuit system removes the pattern image in the case where thestored pattern image is contained in the overlapping image domain and ahuman is undetected from the overlapping image domain.

According to still another aspect of the invention, the human pursuitsystem further includes means that extracts feature amount data from thepattern image, and the extracted feature amount data is used for thepattern matching.

According to still another aspect of the invention, the feature amountdata in the human pursuit system includes at least one of a brightnesspattern, a color histogram, a ratio of color components, an edgehistogram, a Fourier spectrum, and a Fourier-Mellin spectrum.

According to still another aspect of the invention, the human pursuitsystem further includes means that generates a trajectory on which thedetected human moves every shot domain pursued by the plurality ofcameras.

According to still another aspect of the invention, the human pursuitsystem further includes means that merges the generated trajectories forevery detected human.

According to still another aspect of the invention, the plurality ofcameras in the human pursuit system are adjusted respectively such thata shooting angle is directed toward a vertically downward direction.

According to still another aspect of the invention, the human pursuitsystem further includes means that stores attribute information obtainedfor the detected human in association with the trajectory obtained forthe detected human.

According to still another aspect of the invention, the attributeinformation in the human pursuit system is obtained from readingnon-contact radio tag information provided for the human.

Further, the human pursuit apparatus in the invention includes: meansthat obtains images shot by a plurality of cameras installed on aceiling and shooting directions of these directed toward a floor; meansthat calculates a parallax of an object reflected in an overlappingimage domain in accordance with at least a portion of the overlappingimage domain where the images are overlapped, among shot images shotrespectively by the plurality of cameras; means that detects the object,as a human, when the calculated parallax is greater than a predeterminedthreshold value; means that extracts a pattern image containing thedetected human; and means that pursues a trajectory on which the humanmoves, by a pattern matching of the extracted pattern image with theshot image by the camera.

Further, the human pursuit program readable, as program codes, by acomputer includes: program code means that obtains images shot by aplurality of cameras installed on a ceiling and shooting directions ofthese directed toward a floor; program code means that calculates aparallax of an object reflected in an overlapping image domain inaccordance with at least a portion of the overlapping image domain wherethe images are overlapped, among shot images shot respectively by theplurality of cameras; program code means that detects the object, as ahuman, when the calculated parallax is greater than a predeterminedthreshold value; program code means that extracts a pattern imagecontaining the detected human; and program code means that pursues atrajectory on which the human moves, by a pattern matching of theextracted pattern image with the shot image by the camera.

According to the invention, the human movement trajectory can be madepursued in high accuracy even though blocking objects reside in an openspace.

According to the invention, a human object is detected in accordancewith the shot images shot by the plurality of cameras, the shot areas ofwhich are overlapped, and a position of the human object as a positioncoordinate on the floor is obtained from a perspective transformationbased on the parallax, so that the human movement trajectory can be madepursued in high accuracy.

According to the invention, the cameras are adjusted so that theshooting direction is directed toward the floor in the verticallydownward direction, therefore, an error in the perspectivetransformation performed the position of human transformed to acoordination system from the shot image can be made small, and the humanmovement trajectory can be pursued in high accuracy.

According to the invention, since the cameras are installed on theceiling to shoot the floor, a block for a human residing in a distancehardly occurs by causing a human residing in proximity to the camera,and a stereo measurement domain (overlapping image domain) using theparallax is also limited to a narrow range from the ceiling to thefloor, so that the human movement trajectory can be made pursued in highaccuracy.

According to the invention, after a human object is detected by using aportion of the stereo measurement domain (overlapping image domain), thehuman object is pursued by the pattern matching processing using thepattern image of the detected human object in domains other than thestereo measurement domain, so that a processing load can be reduced.

The other objects, features and advantages of the invention will becomeapparent from the following description of the embodiments of theinvention taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a constitutional diagram showing a human pursuit system in anembodiment of the invention;

FIG. 2 is a functional block diagram showing a type-A local device;

FIG. 3 is a functional block diagram showing a type-B local device;

FIG. 4 is a diagram showing an arrangement example of the local devices;

FIG. 5 is a functional block diagram showing a combination example ofthe local devices;

FIG. 6 is a functional block diagram showing an another combinationexample of the local devices;

FIGS. 7A to 7E are diagrams showing output examples from a parallaxmeasuring unit;

FIG. 8A is a processing flow diagram of a threedimensional-suspicious-human object holding unit;

FIG. 8B is a diagram showing an image subject to a mohologicalprocessing;

FIG. 8C is a diagram showing surrounding rectangles overlapped togetherwith pixel groups;

FIG. 9 is a diagram showing an example of a pattern data table held in apattern holding unit;

FIG. 10 is a processing flow diagram of a pattern collating unit;

FIG. 11 is a processing flow diagram of a pattern updating unit;

FIG. 12A is a diagram showing trajectory data generated by a localtrajectory generating unit;

FIG. 12B is a diagram showing an example of a human trajectory;

FIG. 13 is a diagram showing an example of a trajectory mergingprocessing;

FIG. 14 is a processing flow diagram of a trajectory merging unit;

FIG. 15 is a diagram showing an processing example in which overlappinghuman objects are merged and human IDs are edited; and

FIG. 16 is a diagram for explaining output data from the trajectorymerging unit.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, embodiments of the invention will be described withreference to the drawings.

FIG. 1 is a diagram showing a human pursuit system 1 in an embodiment ofthe invention. As shown in FIG. 1, the human pursuit system 1 includes aplurality of local devices 11, 12, . . . 19. Each of the local deviceshas a wide-angle camera installed on a ceiling in a building to bemeasured and adjusted with its shooting angle so as to be facedsubstantially straight to a floor in the building.

Each of the local devices processes in turn every frame of imagesobtained from the camera to pursue a trajectory indicating that anobject (human) reflected by the camera moves on shot areas. Pursuedtrajectory data is transmitted to a human pursuit system managementdevice 30 at every predetermined time period, for example, every onesecond or 30 frames etc. Data transmission/reception between the localdevice and the human pursuit system management device 30 may beperformed by a dedicated signal line, and may also be performed by anetwork using a protocol of TCP/IP etc.

It is assumed that the trajectory data obtained from the local device isrepresented by a coordination system (hereinafter, referred to as afloor coordination system) in which a specific place in a floor domainshot by the camera of the local device is set to an origin. In addition,a position coordinate in the floor coordination system of human objectsdetected by the respective local devices may be computed by a knownperspective transformation processing.

The human pursuit system management device 30 provides with a trajectorymerging unit 301 and a trajectory database 302. The trajectory mergingunit 301 merges the trajectory data, in conjunction with an identicalhuman object, received from the respective local devices at everypredetermined time period. The merged successive trajectory data isstored in the trajectory database 302. This storing operation of thetrajectory data in the trajectory database 302 may be performed at everypredetermined time period.

In this embodiment, there are two types for the local device: type A andtype B.

FIG. 2 shows an example of a type-A local device. The type-A localdevice is installed on a ceiling or an open space in a building, andconstituted by a wide-angle camera 101, the shooting direction of whichis adjusted to be faced substantially straight to the floor, a patternholding unit 102, a pattern collating unit 103, and a local trajectorygenerating unit 104.

The pattern holding unit 102 extracts feature amount data from an imagedomain in which a detected human object is reflected, and holds it. Thefeature amount data to be extracted may be used of various pieces ofinformation, such as, a brightness distribution, a ratio of colorcomponents of red, green, blue, etc., a ratio of respective directionalcomponents in local directions of a human object contour divided intoeight proximity directions, a Fourier spectrum of an image domain, aFourier-Mellin spectrum, etc. The extracted feature amount data isstored in a memory as a pattern data table in association with a humanidentification (ID) to be identified the detected human object.

The human ID is identification information to be given at a time ofstarting a human pursuit performed by each of the local devices. Thehuman ID may be given by numbering the local devices in sequence, and aninherent ID may also be given to each human object by using an ID of thelocal device, a time when the human pursuit starts, and a numbergenerated by the floor coordinate, in the human pursuit system as awhole.

The pattern collating unit 103 performs a pattern matching processingfor the image shot by the wide-angle camera 101 by using the pattern, asa template, turned into table data held in the pattern holding unit 102.As a result of the pattern matching processing, the pattern collatingunit 103 sets a center of most similar domain to a feature of thetemplate, to a new position coordinate of the human object in the floorcoordination system to output the position coordinate to the localtrajectory generating unit 104. Further, the pattern collating unit 103outputs, to the pattern holding unit 102, the feature amount inassociation with brightness pattern data of a pattern image segmentedfrom the images on the basis of the new position coordinate, and acolor, an edge, a spectrum, etc, extracted from the pattern image,together with the position coordinate in the floor coordination system.

The pattern holding unit 102 updates the pattern data table on the basisof the data input from the pattern collating unit 103. Further, thepattern holding unit 102 also accepts inputs of the various featureamount containing the brightness data in first, together with theposition coordinate in association with an object covered by theperspective of wide-angle camera 101, from the pattern collating unit103 in an adjacent local device. The pattern holding unit 102 adds thenew human ID and its feature amount to the pattern data table to updatethis table on the basis of the accepted data.

The local trajectory generating unit 104 generates a movement trajectoryof the human ID stored as table data in the pattern holding unit 102 atevery predetermined time period, such as, one second or an amount of 30frames, to output to the human trajectory system management device 30for a predetermined time interval. In the case of the pattern matchingprocessing performed in the pattern collating unit 103, for example, acorrelation coefficient is calculated from between a scanning image andthe template, and a proximal domain to 1 is then set to a collationresult. The correlation coefficient may be calculated by calculatingcorrelation coefficients of the respective colors of red, green and bluefirst, then adding the respective correlation coefficients with apredetermined weight. Further, the collation method using thecorrelation coefficient may use a known normalization correlation.

FIG. 3 is a diagram showing a functional block example of a type-B localdevice. As shown in FIG. 3, The type-B local device is constituted bythe wide-angle camera 101 faced straight to the floor, a pattern holdingunit 202, a pattern collating unit 203, a local trajectory generatingunit 204, a pattern updating unit 205, athree-dimensional-suspicious-human-object holding unit 206, a parallaxmeasuring unit 207, a three-dimensional-suspicious-human-object holdingunit 208, and a parallax measuring unit 209. Since operations of thepattern holding unit 202, pattern collating unit 203, and localtrajectory generating unit 204 are the same as those of the patternholding unit 102, pattern collating unit 103, and local trajectorygenerating unit 104, respectively, descriptions for those are thereforeomitted.

The parallax measuring units 207, 209 generate a parallax image on thebasis of a parallax measured for the shot area where the image given bythe wide-angle camera 101 is overlapped with an image given by anothercamera adjacent thereto.

Both the units 206, 208 binarize respectively the parallax imagesgenerated from the parallax measuring units 207, 209 by using apredetermined threshold value to thereby hold a center coordinate of asuspicious human object domain concerned with an upper body of the humanand a suspicious human object list having a vertical and horizontal sizeof the suspicious human object domain. In addition, the suspicious humanobject domain may be corresponded to a surrounding rectangle of theupper body image of the human.

The pattern updating unit 205 will be described in detail later. In thecase where a mismatch occurs for the suspicious human object list heldin the units 206, 208, the pattern data table held in the patternholding unit 202, and a position of the human object on pursuit and thevertical and horizontal size of the surrounding rectangle, the patterndata held in the pattern holding unit 202 for the human object onpursuit is updated on the basis of the suspicious human object list heldin the units 206, 208.

Further, a pattern data is added to the pattern data table of the humanobject on pursuit held in the pattern holding unit 202 as a human newlygoing into the shot area when the pattern data is present in thesuspicious human object list held in the units 206, 208, but the patterndata is absent in the pattern data table of the human object on pursuitheld in the pattern holding unit 202. At this time, newly given human IDmay be named by setting an ID of local device, a time when the pursuitof the human starts, and a number having the floor coordinate.

In addition, the respective devices contained in the human pursuitsystem 1 may realize the foregoing functions by reading computerreadable program codes stored in a computer readable recording medium(DVD-ROM, CD-ROM, Flash-memory, etc.) via a medium reading device (notshown).

Here, FIG. 4 shows an arrangement example of the local devices in aretail store as human pursuit system 1. Referring to FIG. 4, shot areascovered by the wide-angle cameras provided respectively in the localdevices are illustrated by circles in an overhead view of the retailstore. In this case, a store entrance is located at the top of overheadview, and a counter is located at the left side thereof. Rectangles withdiagonal lines indicate high-rise display racks, and the top ofhigh-rise display racks and an opposite side of the racks are invisiblefrom the wide-angle camera since the opposite side of the display rackis blocked by itself, even though the display rack is located in thecircle as a shot area covered by the camera, but displaced from thecenter of circle.

In the case of FIG. 4, nine local devices 11, 12, . . . , 19 areinstalled, and the shot areas 111 to 119 covered by the wide-anglecameras contained respectively in the local devices are illustrated bythe circles. Further, overlapping domains caused by the shot areascovered by the adjacent cameras are illustrated by waffle patterns as astereo measuring domain 400.

FIG. 5 is a diagram showing a combination example of the local devicesin the human pursuit system 1. In the case of the combination shown inFIG. 5, the type-B local devices 11, 13 are arranged adjacent to thetype-B local device 12 respectively having the wide-angle camera 101.Basically, the human object is pursued by the pattern matchingprocessing at near the center of shot area covered by the wide-anglecamera. In the case of the overlapping domain of near the periphery shotarea and another shot area covered by the other cameras, the humanobject is pursued with confirmation of a three dimensional configurationby the stereo measurement. In the case of the overlapping domain (stereomeasuring domain) of the shot areas, it is unnecessary to use abackground difference method or an interframe difference method fordetecting and pursuing the human object, so that the detection andpursuit for the human object can be performed appropriately even for aslow moving human and a stationary human.

Further, the combination of the local devices in the human pursuitsystem 1 is not limited to the foregoing examples. As shown in FIG. 6,the type-A local device 14 may be incorporated into a portion of thelocal device. The type-A local device is incorporated into the humanpursuit system 1, so that a load of the parallax measurement can bereduced in the human pursuit system 1 as a whole. The type-A localdevice is installed on a place where the human moves in and out once ina while, so that a processing load can be reduced without deterioratingthe human detecting accuracy in the human pursuit system 1 as a whole.

FIGS. 7A to 7E are explanatory diagrams showing examples of parallaximages output from the parallax measuring units 207, 209. FIGS. 7A and7B show an example of a moving human with a shade. FIG. 7A is anoverhead image shot by the wide-angle camera in the local device, andseen the human from above in FIG. 7B. A camera (not shown) adjacent tothe other also shoots the same human, and the floor coordinate of ahuman standing position can be obtained from the perspectivetransformation. However, as a height increases from the floor level, adisplacement amount of the image increases. Therefore, the displacementincreases in order from portions of legs, abdomen, shoulder and head.FIG. 7C shows a parallax image obtained from changing patterns for theportions of head, shoulder and legs, on the basis of the displacementamount. The portion of shade has the same size as that of the parallaxon the floor, so that an error detection caused by reflection in theshade and floor is prevented by extracting a domain having the parallaxlarger than a predetermined size. FIGS. 7D and 7E will be describedlater.

FIG. 8A is an explanatory diagram showing a processing of the units 206and 208. The units 206 and 208 process the following steps for theparallax image generated from the parallax measuring unit 207 or 209 tothereby update the suspicious human object list.

The unit 206 binarizes the parallax image generated from the parallaxmeasuring unit 207 in comparison with a predetermined threshold value(step S11). The threshold value may be changed in response to a heightof the human object to be detected, for example, setting the thresholdvalue to high is targeted to a tall human, and setting to low istargeted to a not tall human, including adult and children.

Next, a mohological processing (expansion and contraction processing) isgiven to the binarized image to perform a correction processing, such asa link of detached objects, removal of isolated points, filling ofobjects, etc. (step S12). A labeling processing is given to thebinarized image obtained from the mohological processing. The labelingprocessing is performed so that the binarized image is divided intopixel groups in response to a link condition of four or eight proximalpixels, and a label is assigned to every pixel group (step S13). Asurrounding rectangle of the pixel group is then set in response to thedivision by the labeling processing. A center coordinate of thesurrounding rectangle is measured in response to the setting of everylabel, and a vertical and horizontal pixel size is also measured inresponse to the surrounding rectangle to update the suspicious humanobject list (step S14).

FIG. 8B shows an image after applying the mohological processing, andthe domains with diagonal lines show the pixel groups divided by thelabeling processing. FIG. 8C is a diagram illustrating the surroundingrectangles overlapped together with the respective pixel groups relativeto the labeling processing. Here, FIG. 7D shows an image after applyingthe successive processing of FIG. 8A to the parallax image. FIG. 7Eshows an image after applying a mask processing using FIG. 7D as a maskimage to the overhead image in FIG. 7A. Here, brightness information forthe upper body portion having a certain height is only remained, thebrightness information for the portion having no height, such as, thefloor and shade is masked as zero. FIG. 7E shows black where is maskedas zero. In the case of the pattern holding unit to be described later,a domain other than the zero is held therein as a brightness pattern.

FIG. 9 shows an example of the pattern data table held in the patternholding units 102 and 202. The pattern data table is constituted by arecord in association with coordinate information for every human ID,brightness pattern, color histogram, edge histogram, and Fourierspectrum. An identification number managed by the local device is givento the human ID so that a human object on the different trajectory isnot assigned with the same identification number. In addition, in thecase where the same human left the shot area once and comes in that areaagain, a different identification number is assigned to that humanobject. The coordinate information may contain the center coordinate ofthe surrounding rectangle of the respective patterns and the number ofpixels in the vertical and horizontal of the surrounding rectangle. Inthe case of the foregoing example, the pattern data table stores thebrightness pattern, color histogram, edge histogram, and Fourierspectrum, as the feature amount of the pattern image, however, it maystore a portion of these, and may also store other feature amounts.

FIG. 10 is a diagram showing a processing flow of the pattern collatingunits 103 and 203. The human pursuit system 1 is constituted so that aone-cycle processing is performed by inputting every one-frame imagefrom the wide-angle camera 101. The wide-angle camera 101 suppliesone-frame of the shot image to the pattern collating unit 103 (stepS101).

The pattern collating unit 103 searches the pattern data table held inthe pattern holding unit 102 or 202 (step 102). One pattern, forexample, the brightness pattern is then read from a table in the patterndata table if the search is not completed (step S103). The patterncollating unit 103 scans the image supplied with the brightness patternwhich is read, to extract an image domain having the highest correlationwith the brightness pattern (step S104). A knownnormalization-correlation calculation may be used for the calculation ofpattern correlation. In addition, the correlation calculation may alsouse the brightness pattern of Fourier spectrum, color histogram, andedge histogram. The correlation may also be calculated afterstandardization is made by using an average value obtained from thosevalues and a summation of the histograms.

The pattern collating unit 103 sets an image domain indicating that thecalculated correlation is equal to or greater than a predeterminedvalue, to a suspicious human object pattern (step S105). As describedabove, the steps 102 to 105 are repeated, and a verification processingfor the suspicious human object pattern is performed after completingthe search of the pattern data table (step S106). Here, confirmation ismade whether one pattern contains plural suspicious human objectpatterns (step S107). If there are plural suspicious human objectpatterns, one suspicious human object pattern is pointed out eventuallyon the basis of the correlation which is high or low caused by the colorhistogram, edge histogram, and Fourier spectrum (step S108). Bydetermining one suspicious human object pattern eventually, the featureamount is calculated for the brightness pattern, color histogram, edgehistogram, Fourier spectrum, etc. on the basis of its suspicious humanobject domain. The pattern data table is then updated by using thecalculated feature amount (step S109).

The updated pattern data table is output to the pattern holding unit 202(step S110). The human ID and center coordinate among data in thepattern data table are only output to the local trajectory generatingunits 104 and 204. Further, the pattern size may be maintained as aninitial pattern, and may also be changed on the basis of a table whichis formed in response to a perspective position covered by the camera inadvance.

FIG. 11 is a diagram showing a processing flow of the pattern updatingunit 205. First, the patter data table is read from the pattern holdingunit 202 (step S201). The process judges whether there is an unprocessedpattern among the patterns stored in the pattern data table (step S202).If there is the unprocessed pattern (step S202, Y, in this case), theunprocessed pattern is read (step S203). The process judges whether aposition coordinate of the unprocessed patter is resided in anoverlapping domain (step S204). The overlapping domain means the stereomeasuring domain (overlapping image domain) 400 where the shot area 111is overlapped with the shot area 112 shown in FIG. 4. When the processjudges that the pattern is contained in the stereo measuring domain 400,the pattern is collated with a pattern held in the unit 206 (step S205).As a result of the collation, if there is a targeted pattern (step S206,Y, in this case), an updating is applied to the coordinate of patternimage in the pattern data table by the targeted pattern and to thenumber of pixels in the vertical and horizontal surrounding rectangle(step S207). On the contrary, if there is no targeted pattern in thestereo measuring domain 400 (step S206, N, in this case), the pattern inthe pattern data table is deleted (step S208). At the step S204, whenthe process judges that the pattern is not contained in the stereomeasuring domain 400, it returns to the step S202 to repeat the steps.

When the processing is completed for all of the patterns (step S202, N,in this case), the process judges whether there is a pattern in thesuspicious human object which is not collated with the pattern datatable (step S209). If there is a still-uncollated-suspicious humanobject, a new record is added to the pattern data table as a newlydetected human (step S210). The feature amount containing the brightnesspattern etc. is extracted on the basis of the coordinate and the numberof pixels in the vertical and horizontal surrounding rectangle in thepattern data table which is rewritten by the removal, correction andaddition by the foregoing processing (step S211), and the processing inthe pattern updating unit 205 is then completed.

FIGS. 12A and 12B are diagrams showing an example of data output fromthe local trajectory generating units 104 and 204 to the trajectorymerging unit 301. The local trajectory generating units 104 and 204transmit the position coordinates of human object on the predeterminednumber of frames (for example, 30 frames) with the human ID set to anindex for every predetermined time period (for example, one second). Thetrajectory merging unit 301 links the position coordinates of the samehuman object on the respective frames, as a linear or curved line, toobtain a movement trajectory of the human object.

In the case of FIGS. 12A and 12B, it is assumed that the human is movingin the shot area covered by the wide-angle camera 101. As describedabove, in the case where the human left the shot area once and comes inthat area again, a new human ID is assigned to that human object eventhough it is the same. In FIGS. 12A and 12B, the human ID are numberedby 1 to 12, however, may also be indicated by an ID of the camera, atime when the trajectory is detected at the beginning, a frame number, aunique value generated from a coordinate value on the floor coordinationsystem of the firstly detected position.

FIG. 12A shows an example of trajectory data generated by the localtrajectory generating units 104, 204. The trajectory data contains theposition coordinates on the 30 frames containing every human ID, and istransmitted to the human pursuit system management device 30 for everypredetermined time interval (time s1, s2, . . . ). The trajectory datain FIG. 12 indicates that the data transmitted at every time is, inturn, accumulated.

FIG. 12B is a diagram representing an example of a human trajectorywhich is made visible on the basis of the foregoing transmittedtrajectory data. In FIG. 12B, the trajectories for the human ID “1”, “2”and “12” are shown on the basis of the trajectory data transmitted at atiming of s1.

FIG. 13 shows a trajectory aspect drawn by the human pursuit systemmanagement device 30 on the basis of the trajectory data which iscomposed of a single unit containing 30 frames generated by the localdevice 11, when the trajectory data is transmitted by four times. Thehuman pursuit system management device 30 transforms the human IDs givenby the respective local devices into a format containing such “the humanIDs given by the local devices” plus “IDs of the local devices.”Assuming that the respective IDs for the local devices 11, 12, . . . ,19 are set to CAM1, CAM2, . . . , CAM19, the human ID “12” of localdevice 11 is transformed into “12CAM1.” The trajectory merging unit 301in the human pursuit system management device 30 links the respectivecoordinate values contained in the trajectory data to be transmittedwith the same human IDs, as described above, to thereby form a string oftrajectory.

By the foregoing trajectory data communication processing and trajectorymerging processing, the movement trajectory of each human object isgenerated for the human resided in the shot area covered by thewide-angle camera 101 belonging to each local device.

Referring to FIGS. 14 to 16, by merging the human movement trajectory inthe shot area covered by the local device, a processing performed by thehuman pursuit system management device 30 will be described below.

FIG. 14 is a diagram showing a processing flow of the trajectory mergingunit 301. First, the human pursuit system management device 30 receivesthe trajectory data generated from the local trajectory generating unitin each local device (step S301). It is assumed that each trajectorydata is represented by the floor coordination system.

Next, the human pursuit system management device 30 generates atrajectory for each shot area covered by each local device on the basisof each trajectory data which is received. Until the processing iscompleted for the trajectory data of the predetermined single unit (forexample, 30 frames) (step S302, N, in this case), the human ID is editedby setting the trajectory having an overlapping or proximity portion tothat for the same human object, among the generated trajectories (stepS303). In this way, when the processing is completed for the trajectorydata of the predetermined single unit (step S302, Y, in this case), thetrajectory data in which the human ID is edited is saved in thetrajectory database 302 (step S304). The trajectory merging unit 301repeats the foregoing processing at every time of receiving thetrajectory data.

FIG. 15 is a diagram for explaining a merging processing for theforegoing human objects and trajectories. In FIG. 15, each human objecttrajectory is illustrated on the basis of the received trajectory data.In the case of an example shown in FIG. 15, the trajectories areillustrated on the basis of the trajectory data generated by the localdevice 11 at a time k−1 to a time k and the trajectory data generated bythe local device 14 at the time k to a time k+2.

The trajectory of the human ID “12CAM1” is approximated to that of thehuman ID “34CAM4” at the time k. In the case where the trajectories areclose to one another, it is determined that the trajectories arebelonged to the same human object, consequently, the human objects andtrajectories are merged. Whether the trajectories are close to oneanother is judged by a case where a distance between the floorcoordinates of trajectories at the time k is measured, and the distancebetween the trajectories is continuously remained within a predeterminedrange (for example, within 40 cm). In this case, the trajectories aredetermined that “the trajectories are close to one another.” Thepredetermined range may be changed in response to circumstances wherethe human pursuit system is installed. For example, the predeterminedrange may be made smaller in the case where a lens distortion of thewide-angle camera in the local device is small and a measuring accuracyfor the human position coordinate is high.

FIG. 16 is a diagram for explaining merged trajectory data representingthe trajectory merged by the trajectory merging unit 301. The mergedtrajectory data is data containing a reception time, a time, trajectorydata generated by the local trajectory generating unit in each localdevice, and trajectory merged information representing the trajectoriesto be merged and its time.

Referring to FIG. 16, periods during which the position coordinates areobtained for the human IDs “11CAM1”, “12CAM1”, “33CAM4”, and “34CAM4”are indicated by lines having arrows at both ends. The foregoing mergedtrajectory data merged by the trajectory merging unit 301 is aconclusively outputted result obtained by the human pursuit system 1 ofthe invention. According to the foregoing output result, the humanmovement trajectory is output substantially for the measuring open spacein real time.

According to the human pursuit system 1 in the embodiment describedabove, the human movement trajectory can be pursued in high accuracyeven in the open space where the blocking objects are resided therein.Further, it is possible to know, substantially in real time, a movingdirection, a moving speed, a staying time, etc. of every human object inaccordance with the movement trajectory of every human body. Therefore,the human pursuit system 1 can be used for a security system by giving amark to a suspicious human object on a display screen in response to theforegoing conditions and also giving a warning to an outside.

Further, the human pursuit system 1 in the embodiment can be used in anopen space other than retail stores since devices such as non-contact ICtag etc. are not used. The human pursuit system 1 can also be installedin a building where the ceiling is low level since the camera is facedto the floor to hardly occur a blocking against a human resided in adistance, caused by a human resided close to the camera.

Further, the human pursuit system 1 in the foregoing embodiment is notsustained from infringement of individual privacy since each of thehuman movement trajectory is measured and recorded without specifyingthe human. In addition, the foregoing embodiment is applicable also inthe case where an individual is specified to pursue its movementtrajectory, because it is possible to measure and record easily theindividual movement trajectory in association with specified userinformation by giving a non-contact radio tag to each human, byinstalling a radio tag reader in a specific area, and by reading thenon-contact radio tag of each human at points, such as an entrance orexit of the specific area.

In addition, the invention is not limited to the foregoing embodiment.For example, in the embodiment, the trajectory merging unit 301 enablesthe output data to be stored in the trajectory database 302, whichenables old trajectory data to reproduce itself. The human pursuitsystem 1 of the invention can be used for a marketing system formeasuring movement paths of customers and the number of incomingcustomers in the retail store, and also for a security system forloiterers who stay in too long or walk back and forth in the store.

Furthermore, the human pursuit system 1 in the embodiment is applicableeasily to a customer information collecting system for collectinginformation of the customer types since a number of the human movementtrajectories are measured continuously, and path information in responseto the gender and age of the customers can be obtained by once enteringattributes the gender, age, etc. of the customers from a visual judgmentby a shop clerk or an automatic judging device at the counter located atthe exit or entrance of the store.

It should be further understood by those skilled in the art thatalthough the foregoing description has been made on embodiments of theinvention, the invention is not limited thereto and various changes andmodifications may be made without departing from the spirit of theinvention and the scope of the appended claims.

1. A human pursuit system, comprising: a plurality of cameras installedon a ceiling and shooting directions of these directed toward a floor;means that calculates a parallax of an object reflected in anoverlapping image domain in accordance with at least a portion of theoverlapping image domain where images are overlapped, among shot imagesshot respectively by the plurality of cameras; means that detects theobject, as a human, when the calculated parallax is equal to or greaterthan a predetermined threshold value; means that extracts a patternimage containing the detected human object; and means that pursues atrajectory on which the human object moves, by a pattern matching of theextracted pattern image with the shot image by the camera.
 2. The humanpursuit system according to claim 1, further comprising calculationmeans that calculates a position coordinate of the human object residingon the floor in accordance with a position of the pattern imagecontaining the detected human object and the parallax calculated for thehuman object.
 3. The human pursuit system according to claim 1, furthercomprising: storage means that stores the extracted pattern image forevery detected human object; and update means that updates the patternimage stored in the storage means in accordance with a result of thepattern matching.
 4. The human pursuit system according to claim 3,wherein the storage means further stores the position coordinate of thepattern image on the floor in association with the pattern image, andthe update means updates the pattern image in accordance with an imagedomain of the human object detected from the overlapping image domain ina case where the stored pattern image is contained in the overlappingimage domain.
 5. The human pursuit system according to claim 4, whereinthe update means removes the pattern image in a case where the storedpattern image is contained in the overlapping image domain and the humanobject is undetected from the overlapping image domain.
 6. The humanpursuit system according to claim 1, further comprising means thatextracts feature amount data from the pattern image, wherein theextracted feature amount data is used for the pattern matching.
 7. Thehuman pursuit system according to claim 6, wherein the feature amountdata includes at least one of a brightness pattern, a color histogram, aratio of color components, an edge histogram, a Fourier spectrum, and aFourier-Mellin spectrum.
 8. The human pursuit system according to claim7, further comprising means that generates a trajectory on which thedetected human object moves every shot domain pursued by the pluralityof cameras.
 9. The human pursuit system according to claim 8, furthercomprising means that merges the generated trajectories for everydetected human object.
 10. The human pursuit system according to claim1, wherein the plurality of cameras are adjusted respectively such thata shooting angle is directed toward a vertically downward direction. 11.The human pursuit system according to claim 1, further comprising meansthat stores attribute information obtained for the detected human objectin association with the trajectory obtained for the detected humanobject.
 12. The human pursuit system according to claim 11, wherein theattribute information is obtained from reading non-contact radio taginformation provided for the human object.
 13. A human pursuit system,comprising: means that obtains images shot by a plurality of camerasinstalled on a ceiling, and shooting directions of these directed towarda floor; means that calculates a parallax of an object reflected in anoverlapping image domain in accordance with at least a portion of theoverlapping image domain where the images are overlapped, among shotimages shot respectively by the plurality of cameras; means that detectsthe object, as a human, when the calculated parallax is equal to orgreater than a predetermined threshold value; means that extracts apattern image containing the detected human object; and means thatpursues a trajectory on which the human object moves, by a patternmatching of the extracted pattern image with the shot image by thecamera.
 14. A non-transitory computer-readable medium embodying a humanpursuit program readable, as program codes, by a computer, comprising:program code means that obtains images shot by a plurality of camerasinstalled on a ceiling, and shooting directions of these directed towarda floor; program code means that calculates a parallax of an objectreflected in an overlapping image domain in accordance with at least aportion of the overlapping image domain where the images are overlapped,among shot images shot respectively by the plurality of cameras; programcode means that detects the object, as a human, when the calculatedparallax is equal to or greater than a predetermined threshold value;program code means that extracts a pattern image containing the detectedhuman object; and program code means that pursues a trajectory on whichthe human object moves, by a pattern matching of the extracted patternimage with the shot image by the camera.